TEST  AND  EVALUATION 


Paving  the  Way  for 
Testing  in  a  Joint  Environment 

The  Capability  Test  Methodology 


Chris  DiPetto 


* 


The  very  nature  of 
modern  warfare 
necessitates  major 
changes  to  the  way 
the  Department  of 
Defense  tests  and  acquires 
systems  and  capabilities. 
Since  most  systems  today 
are  deployed  in  joint  envi¬ 
ronments,  the  testing  of  a 
system  by  a  single  Service 
may  not  be  adequate  to 
demonstrate  that  the  sys¬ 
tem  meets  the  warfighter’s 


As  DoD  moves  away  from 
traditional  single-system 
approaches  to  new  joint 
capability-based  approaches, 
the  department  must 
demonstrate  that  its  future 
weapons  integrate  seamlessly 
into  systems  of  systems  and 
capabilities  in  complex 
battlespace  environments  to 
produce  coordinated 
and  focused  effects. 


fight.  The  warfighter 
should  be  confident 
the  systems  work  as 
advertised,  and  the 
tester  must  be  chal¬ 
lenged  to  deliver  the 
future  joint  capabili¬ 
ties  needed  by  the 
warfighter. 


needs.  Future  systems, 

which  are  expected  to  operate  in  a  joint  environment, 
should  be  tested  in  a  realistic  joint  environment  through¬ 
out  the  acquisition  life  cycle,  starting  with  early  experi¬ 
mentation  and  concept  development  through  the  devel¬ 
opmental  and  operational  test.  The  result  is  an  optimally 
integrated  system. 

DoD  needs  to  ensure  that  it  is  testing  systems,  systems 
of  systems  (SoS),  and  capabilities  consistent  with  their  in¬ 
tended  use.  In  other  words,  we  test,  as  well  as  train,  like  we 


A  New 
Roadmap 

Changes  in  testing  and 
acquisition  processes  are 
under  way  to  make  this 
happen.  DoD  instructions 
acknowledge  the  need  to 
test  joint  capabilities  in  the 
expected  joint  operational 
environment.  The  Joint 
Capabilities  Integration 
and  Development  System 
(JCIDS)  is  applying  capa- 
bilities-based  approaches 
to  transform  the  way  the 
DoD  defines  requirements 
for  new  systems  and  ca¬ 
pabilities  by  moving  mate¬ 
riel  developers  and  testers 
away  from  the  Service-cen¬ 
tric  system  requirements 
of  the  past  and  toward  the 
necessary  joint-centric  capability  development  for  future 
systems.  In  November  2004,  in  response  to  strategic  plan¬ 
ning  guidance  direction  to  provide  new  testing  capabilities 
and  institutionalize  the  evaluation  of  joint  system  effec¬ 
tiveness  as  part  of  new  capabilities-based  processes,  then- 
Deputy  Secretary  of  Defense  Paul  Wolfowitz  approved 
the  Testing  in  a  Joint  Environment  Roadmap,  developed 
by  the  DoD  director  of  operational  test  and  evaluation. 
The  roadmap  calls  for  actions  that  establish  a  framework 
for  the  life  cycle  evaluation  of  systems  and  SoS  in  a  joint 
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operational  environment  beginning  with  the  JCIDS  pro¬ 
cess.  Implementation  of  the  roadmap  focuses  on  three 
elements— policy,  infrastructure  networks  and  middle¬ 
ware,  and  methods  and  processes— while  recognizing  the 
important  role  of  the  Services  and  agencies  in  the  execu¬ 
tion.  The  purpose  of  the  roadmap  is  to  coordinate  and 
synchronize  the  sometimes  disparate  Service  and  agency 
testing  efforts  by  capitalizing  on  existing  Service  test  as¬ 
sets,  forming  an  approach  to  joint  testing  that  will  ensure 
systems  and  capabilities  function  as  intended  when  inte¬ 
grated  into  the  joint  mission  environment. 

As  individual  platforms  become  part  of  a  complex,  net¬ 
worked  SoS  that  must  work  effectively  in  a  joint  bat- 
tlespace,  effective  test  and  evaluation  is  becoming  more 
difficult.  In  the  future,  programs  expected  to  operate  in  a 
joint  environment  should  demonstrate  their  joint  capa¬ 
bility  early  and  throughout  their  developmental  cycles, 
regardless  of  the  program’s  acquisition  category.  The 
objective  of  the  roadmap  is  to  address  these  challenges 
and  define  changes  that  will  position  test  capabilities  to 
fully  support  adequate  test  and  evaluation  of  warfighting 
capabilities  developed  under  DoD’s  capabilities-based  pro¬ 
cesses  in  the  appropriate  joint  mission  environment. 

The  joint  mission  environment  provides  the  operational 
context  in  which  the  capability  being  developed  must 
perform.  Important  aspects  of  this  operational  context  in¬ 
clude  joint  mission,  task,  threat  condition,  environmental 
condition,  and  system  or  SoS  descriptions  of  capabilities 
supporting  the  joint  mission.  The  joint  mission  environ¬ 
ment  is  realized  when  all  relevant  aspects  of  the  joint 
operational  context  are  adequately  represented  in  an  en¬ 
vironment  ready  for  a  test  that  may  be  live,  virtual,  and / 
or  constructive  and  distributed  in  nature. 

The  Test  Resource  Management  Center,  established  as  a 
field  activity  reporting  to  the  Office  of  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology  and  Logistics,  is 
chartered  to  be  the  steward  of  DoD  test  and  evaluation 
infrastructure.  In  that  capacity  TRMC  is  responsible  for 
the  Joint  Mission  Environment  Test  Capability  Program 
to  help  establish  a  DoD-wide  live,  virtual,  constructive  dis¬ 
tributed  environment  (LVC-DE).  To  date,  JMETC  has  laid 
the  groundwork  for  an  enterprise-level  solution  to  testing 
infrastructure  and  has  enabled  that  infrastructure  to  be 
assembled  more  quickly  than  in  the  past.  Even  with  an 
effective  infrastructure  in  place,  there  is  also  a  need  for 
policy  changes  and  new  methods  and  processes  to  make 
valid  testing  in  a  joint  mission  environment  a  reality. 

The  Methods  and  Processes 

Traditional  methodologies  for  developing  and  testing 
military  systems  expected  to  operate  in  a  joint  environ¬ 
ment — particularly  verifying  specification  compliance 
for  individual  systems  and  testing  within  a  single  Service 
environment — will  fail  to  fully  describe  real  joint  capabili¬ 


ties.  As  DoD  moves  away  from  traditional  single-system 
approaches  to  new  joint  capability-based  approaches,  the 
department  must  demonstrate  that  its  future  weapons 
integrate  seamlessly  into  SoS  and  capabilities  in  complex 
battlespace  environments  to  produce  coordinated  and  fo¬ 
cused  effects.  In  addition,  doctrine,  organization,  training, 
leadership,  personnel,  and  facility  aspects  of  these  new 
capabilities  along  with  materiel  needs  must  be  addressed 
early  in  the  development  process.  DoD’s  long-term  strat¬ 
egy  calls  for  evaluations  of  joint  systems  effectiveness 
throughout  all  phases  of  a  capability’s  development  and 
deployment.  As  I  mentioned  before,  we  want  to  be  able 
to  test  as  we  fight. 

The  challenge  of  testing  in  a  joint  mission  environment 
begins  early  in  the  system  acquisition  life  cycle;  and  it  is 
daunting  considering  the  number  of  systems,  network 
combinations  and  interactions,  environmental  conditions, 
and  non-materiel  aspects  that  must  be  addressed  for  a 
realistic  test.  How  much  testing  and  data  are  sufficient? 
Must  every  possible  combination  of  environmental  condi¬ 
tions,  modes  of  operation,  systems,  and  entities  within 
the  joint  mission  environment  be  exercised?  Replicating 
a  realistic  joint  environment  will  be  very  challenging  be¬ 
cause  the  ability  to  assemble  all  required  assets  at  a  single 
test  location  will  be  nearly  impossible  because  of  sched¬ 
uling  constraints  and  resource  availability.  How  much  of 
this  environment  needs  to  be  available  for  developmental 
testing  and  how  much  is  required  for  operational  testing? 
How  can  a  realistic  joint  environment  be  constructed  to 
enable  it  to  meet  both  developmental  and  operational 
test  objectives?  What  kinds  of  tests  can  be  done  during 
developmental  testing  to  reduce  the  risk  of  uncovering 
new  system  deficiencies  during  operational  testing? 

To  address  some  of  these  challenges  as  part  of  the  larger 
roadmap  effort,  the  director  of  operational  test  and  evalu¬ 
ation  chartered  the  Joint  Test  and  Evaluation  Methodology 
project  in  2006.  Specifically,  JTEM  was  directed  to  develop, 
test,  and  evaluate  methods  and  processes  for  defining  and 
using  an  LVC-DE  joint  test  environment  to  evaluate  system 
performance  and  joint  mission  effectiveness. 

JTEM  has  developed  the  capability  test  methodology, 
which  is  a  collection  of  recommended  best  practices  for 
designing  a  test  of  a  system  or  SoS  in  a  complex  joint 
environment.  The  CTM  provides  a  rational  process  that 
guides  the  program  manager  and  test  manager  through 
the  test  planning  process  to  tailor  and  optimize  a  test  to 
demonstrate  system  performance  within  a  joint  context 
as  well  as  system  contribution  to  joint  capabilities.  It  is  a 
foundation  for  a  series  of  guides,  handbooks,  and  training 
courses  that  will  ultimately  be  delivered  to  test  organiza¬ 
tions  and  acquisition  PMs.  The  CTM  is  intended  to: 

■  Address  testing  of  systems,  SoS,  and  capabilities,  be 
they  Service  or  joint 

■  Augment  existing  DoD  and  Service  test  processes 
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■  Align  test  and  evaluation  aspects- 
and  information  across  multiple 
DoD  processes,  namely  Analytic 
Agenda,  JCIDS,  DoD  Architecture 
Framework,  and  the  Defense 
Acquisition  System 

■  Provide  recommended  best 
practices  for  a  consistent 
approach  to  describing, 
building,  and  using  an  ap¬ 
propriate  representation  of 
a  particular  joint  mission 
environment  across  the 
acquisition  life  cycle 

■  Reflect  current  acquisition 
policies  and  instructions, 
and  eventually  be  incorpo¬ 
rated  into  Defense  Acquisi¬ 
tion  University  PM  and  test  and  evaluation  courses. 

The  CTM  is  designed  to  augment,  not  replace,  existing  test 
methods  and  processes,  taking  into  account  the  unique 
aspects  of  testing  joint,  networked  systems  in  an  LVC-DE. 
As  such,  the  CTM  closely  parallels  existing  test  processes 
used  within  DoD.  The  CTM  consists  of  six  steps  and  14 
processes,  which  are  briefly  described  in  the  following 
paragraphs. 

The  CTM  Steps 

Step  0  defines  the  test  evaluation  strategy.  The  key  pro¬ 
cess  in  this  step  is  describing  the  Joint  Operational  Con¬ 
text  for  Test  used  to  define  the  specific  elements  that 
make  up  the  LVC-DE.  The  Joint  Operational  Context  for 
Test  includes  a  detailed  description  of  the  system  under 
test,  supporting  systems,  the  expected  operating  envi¬ 
ronment,  threat  forces,  and  key  system  interactions  and 
information  exchanges  required  to  complete  a  particular 
task  or  mission.  In  step  1 ,  the  PM  creates  a  program  in¬ 
troduction  document,  which  outlines  the  details  of  a  par¬ 
ticular  test  or  set  of  tests,  communicating  requirements 
to  a  test  range.  The  test  range  then  uses  that  document 
to  produce  a  statement  of  capability,  which  is  the  start¬ 
ing  point  for  determining  what  resources  will  be  used  to 
conduct  the  test  and  what  data  will  be  collected.  Step  2 
produces  distributed  test  plans  that  are  compilations  of 
current  individual  test  plans,  with  the  addition  of  distrib¬ 
uted  and  joint  elements.  During  this  test  planning  phase, 
early  test  concepts  are  developed  into  more  detailed  test 
plans.  Test  planning  processes  include  test  trial/vignette 
selection,  refining  the  live,  virtual,  constructive  distributed 
test  environment  required  and  synthesizing  these  activi¬ 
ties  into  a  test  plan. 

During  CTM  steps  3,  4,  and  5,  joint  mission  environments 
are  assembled  and  used  to  support  multiple  test  plans. 
Step  3  is  concerned  with  technical  systems  engineering 
activities  for  automatic  distributed  LVC-DE  implementa- 
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test  methodology  is 
addressing  the  ways 
and  means  for  designing  and 
executing  tests  of  complex, 
networked  SoS  in  a  realistic  joint 
mission  environment  through  its 
newly  developed  and  enhanced 
methods  and  processes. 


tion.  These  processes  include  the  design  of  distributed 
configurations,  the  assembly  of  distributed  components, 
and  the  integration  of  components  into  a  distributed  test 
range  that  meets  customer  requirements.  In  CTM  step  4, 
distributed  tests  are  conducted  according  to  local  proce¬ 
dures  and  data  are  collected.  This  phase  produces  test 
data  for  customers  and  reusable  information  for  future 
joint  mission  environments.  Though  joint  mission  envi¬ 
ronments  are  assembled  to  support  multiple  customers, 
tests  are  not  required  to  run  concurrently.  Sometimes  in¬ 
dividual  customers  may  separately  schedule  only  those 
parts  of  the  joint  mission  environment  they  need  to  meet 
their  own  objectives  for  testing  in  a  joint  environment. 
Other  times,  multiple  customers  may  share  a  joint  mis¬ 
sion  environment  at  the  same  time  for  convenience  or 
other  reasons.  In  step  5,  data  sets  are  processed,  ana¬ 
lyzed,  and  evaluated  (including  evaluations  of  joint  mis¬ 
sion  effectiveness  and  contributions  of  individual  systems 
to  joint  missions). 

Supporting  Measures  Framework 

In  addition  to  the  CTM,  JTEM  has  developed  a  supporting 
measures  framework  that  establishes  appropriate  mea¬ 
sures  to  support  the  evaluation  of  a  system  or  SoS  within 
a  capabilities  context.  This  framework  is  based  on  the 
JCIDS  definition  of  a  capability:  “The  ability  to  achieve  a 
desired  effect  under  specified  standards  and  conditions 
through  a  combination  of  ways  and  means  to  perform  a 
set  of  tasks.”  Measures  of  effectiveness  are  established  at 
the  mission  level  and  based  on  combatant  commander- 
desired  joint  mission  effects.  The  joint  mission  effects 
are  documented  through  a  compilation  of  products  that 
make  up  the  Analytic  Agenda,  which  is  a  DoD-wide  frame- 
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work  for  analyzing  force  structure  requirements  and  other 
analytic  studies.  The  products  used  to  document  desired 
effects  include  the  defense  planning  scenarios,  which  are 
a  series  of  scenarios  that  describe  the  range  of  military 
operations  for  which  combatant  commanders  must  be 
prepared,  along  with  the  operating  forces  and  threats  de¬ 
scribed  in  the  multi-Service  force  deployment  database 
and  the  current  year  and  future  year  analytical  baselines. 
The  desired  effects  must  be  achieved  under  specified  stan¬ 
dards  and  conditions  using  systems;  SoS;  and  the  support¬ 
ing  doctrine,  organization,  training,  leadership,  personnel, 
and  facility  aspects,  which  make  up  the  combinations  of 
ways  and  means. 

The  systems  and  SoS  have  various  performance  attributes 
associated  with  them  (e.g.,  launch  range  of  an  aircraft  or 
time  to  disseminate  information  to  the  battlefield  from 
a  higher  echelon  headquarters),  and  they  are  ultimately 
are  used  to  perform  a  set  of  joint  tasks  that  achieve  the 
joint  mission  desired  effects.  In  the  measures  framework, 
measures  of  performance  are  used  to  describe  the  overall 
performance  desired  for  each  particular  task.  The  joint 
tasks  are  described  through  the  Universal  Joint  Task  List 
and  the  Joint  Mission  Essential  Task  List,  along  with  the 
specified  standards  and  conditions.  The  Universal  Joint 
Task  List  and  the  Joint  Mission  Essential  Task  List  also 
have  corresponding  Service  task  lists  that  support  them. 
Although  mission  measures  of  effectiveness  will  be  diffi¬ 
cult  to  capture  directly  during  tests  in  a  joint  environment, 
the  task-level  measures  of  performance  and  the  system 
and  SoS  attributes  can  be  readily  measured.  Analysis  and 
combat  modeling  can  then  be  used  to  determine  overall 
measures  of  effectiveness  for  the  joint  mission  desired 
effects. 

Testing  in  a  Realistic  Environment 

The  CTM  is  addressing  the  ways  and  means  for  design¬ 
ing  and  executing  tests  of  complex,  networked  SoS  in 
a  realistic  joint  mission  environment  through  its  newly 
developed  and  enhanced  methods  and  processes.  Dur¬ 
ing  their  2007  test  event,  JTEM  demonstrated  application 
of  the  CTM  to  a  notional  set  of  network-enabled  air-  and 
ground-launched  weapon  systems  while  employed  in  a 
joint  mission  environment  supporting  a  joint  fire  support 
task.  This  test  showed  the  potential  that  can  be  realized 
from  testing  networked  SoS  in  a  realistic  operational  en¬ 
vironment.  However,  it  also  revealed  many  challenges, 
which  fall  into  the  categories  of: 

■  Agreed-to  measures  of  performance  and  effectiveness 
across  multiple  joint  missions 

■  Persistence  of  the  test  environment  used  for  testing 

■  Analysis  and  data  management  techniques  to  deal 
with  the  increasing  complexity  of  planning  tests  and 
evaluating  the  results  of  tests  in  net-centric  systems. 

JTEM  continues  to  address  these  issues.  In  this  year’s  test 
event,  the  CTM’s  effectiveness  and  suitability  for  use  in  a 


complex  joint  test  environment  was  again  assessed  using 
the  Army’s  Combined  Test  Organization  for  Future  Com¬ 
bat  Systems’  Joint  Battlespace  Dynamic  Deconfliction 
event  as  a  joint  capability  test  event.  This  venue  provided 
the  opportunity  to  identify  the  challenges  in  integrating 
the  end-to-end  CTM  into  existing  test  activity,  developing 
and  maturing  the  LVC  distributed  prototype,  investigating 
data-requirements  issues,  analyzing  deficiencies  in  the 
joint  mission  environment  representation  requirements, 
and  assessing  the  LVC  distributed  environment  instantia¬ 
tion  of  the  joint  mission  environment.  Test  event  results 
are  driving  improvements  to  the  CTM  and  will  provide 
an  opportunity  to  gain  a  better  understanding  of  what  it 
takes  to  fully  realize  a  sufficient  capability  to  test  in  a  joint 
environment  across  DoD.  In  April  2009,  JTEM  will  deliver 
a  version  of  the  CTM,  along  with  guides,  handbooks,  and 
additional  supporting  documents,  which  will  prepare  PM 
and  test  organization  customers  to  effectively  test  as  the 
capabilities-based  approach  to  acquisition  requires. 

Continuous  Learning  Available 

Additional  work  has  been  accomplished  to  facilitate  our 
future  testing  needs.  The  Defense  Acquisition  University 
and  JTEM  have  partnered  to  develop  a  Testing  in  a  Joint 
Environment  continuous  learning  module,  now  available 
on  the  DAU  Web  site  <  http://clc.dau.mil/  >  .  The  module’s 
goal  is  to  familiarize  DoD  personnel  with  basic  principles 
and  practices  related  to  testing  in  a  joint  environment. 
This  three-hour  credit  course  will  enable  capability  manag¬ 
ers,  PMs,  requirements  managers,  systems  engineers,  test 
and  evaluation  professionals,  acquisition  professionals, 
and  warfighters  to: 

■  Recognize  the  need  for  testing  in  a  joint  environment 

■  Describe  the  key  DoD-level  concepts  that  support 
testing  in  a  joint  environment 

■  Describe  the  generalized  methodology  for  testing  in  a 
joint  environment 

■  Define  the  structured  approach  for  identifying  mea¬ 
sures  that  support  testing  in  a  joint  environment 

■  Recognize  the  features  of  the  joint  mission  environ¬ 
ment. 

The  future  of  testing  in  a  joint  environment  has  many 
challenges  and  many  exciting  opportunities.  Through  the 
work  of  the  Testing  in  a  Joint  Environment  Roadmap  and 
the  efforts  of  all  in  the  testing  and  acquisition  communi¬ 
ties,  the  challenges  will  be  met  and  the  opportunities  will 
be  exploited.  This  collective  effort  will  enable  us  to  test  like 
we  fight  and  deliver  the  future  joint  capabilities  needed 
by  the  warfighter. 


Comments  or  questions?  Please  go  to  <www.acq. 
osd .  mil /sse  /  dte  /  > . 
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